JBoss Community Archive (Read Only)

Infinispan 6.0

Contributing - The test suite

Tests are written using the TestNG framework.

Running the tests

Before running the actual tests it is highly recommended to configure adjust suite's memory setting by updating the MAVEN_OPTS variables. E.g.

$ export MAVEN_OPTS="-Xms512m -Xmx2048m -XX:MaxPermSize=384m"

The default run executes all tests in the functional and unit groups. To just run the tests with txt and xml output the command is:

$ mvn test

Alternatively, you can execute the tests and generate a report with:

$ mvn surefire-report:report

If you are running the tests on a Unix-like operating system, the default limits per user are typically low. The Infinispan test suite creates a lot of processes/threads, thus you will have to increase your user's limits and reboot the system to pick up the new values. Open up /etc/security/limits.conf and add the following lines replacing the user name with your username.

rhusar    soft    nofile    16384
rhusar    hard    nofile    16384
rhusar    soft    nproc     16384
rhusar    hard    nproc     16384

Specifying which tests to run

A single test can be executed using the test property. The value is the short name (not the fully qualified package name) of the test. For example:

$ mvn -Dtest=FqnTest test

Alternatively, if there is more than one test with a given classname in your test suite, you could provide the path to the test.

$ mvn -Dtest=org/infinispan/api/MixedModeTest test

Alternatively, you can always pass your own Log4j configuration file via -Dlog4.configuration with your own logging settings.

Patterns are also supported:

$ mvn -Dtest=org/infinispan/api/* test

Skipping the test run

It is sometimes desirable to install the Infinispan package in your local repository without performing a full test run. To do this, simply use the maven.test.skip.exec property:

$ mvn -Dmaven.test.skip.exec=true install

Note that you should never use -Dmaven.test.skip=true since modules' test classes depend on other module test classes, and this will cause compilation errors.

Running tests using @Parameters

If you want to execute tests relying on TestNG's @Parameters from an IDE (such as Eclipse or IntelliJ IDEA), please check this blog entry.

Enabling TRACE in test logs

When you run tests, you can get TRACE logging via using the traceTests profile

mvn test -PtraceTests

Executing this will generate a GZIP file called trace-infinispan.log.gz. This file is not fully closed, so to extract the log file, execute:

gunzip -c trace-infinispan.log.gz > trace-infinsipan.log

Enabling code coverage generation

When you run tests, you can enable code coverage recorder for calculating and analysing the Infinispan code coverage. You can do this using coverage and jacocoReport profiles.
As a code coverage evaluation tool, the JaCoCo is used.

mvn test -Pcoverage -Dmaven.test.failure.ignore=true

Please note, that -Dmaven.test.failure.ignore=true is used for generating JaCoCo code coverage report, even if there are test failures.

Executing this will generate jacoco.exec file in each module's target/ directory. These are the JaCoCo execution data files, which contain full data about the specific module's coverage.

As soon as the coverage execution command is finished, you will need to generate the JaCoCo report, which will merge the generated jacoco.exec files as well as will create the code coverage report.

For having the report in place, run the following command from your Infinispan home directory:

mvn install -pl parent -PjacocoReport

The jacoco-html/ directory will be generated in Infinispan Home directory, which will contain the code coverage report.

Test groups

Each test should belong to one or more group. The group acts as a filter, and is used to select which tests are ran as part of the maven test lifecycle.

Which group should I use?

If your test does not fit into one of these group, a new group should be added.

unit

Unit tests using stubs to isolate and test each major class in Infinispan. This is the default group run if no test group is specified

functional

Tests which test the general functionality of Infinispan

jgroups

Tests which need to send data on a JGroups Channel

transaction

Tests which use a transaction manager

profiling

Tests used for manual profiling, not meant for automated test runs

manual

Other tests that are run manually

Every test (except those not intended to be run by Hudson) should at least be in the functional or unit groups, since these are the default test groups executed by Maven, and are run when preparing a release.

Test permutations

We use the term permutation to describe a test suite execution against a particular configuration. This allows us to test a variety of environments and configurations without rewriting the same basic test over and over again. For example, if we pass JVM parameter -Dinfinispan.test.jgroups.protocol=udp test suite is executed using UDP config.

$ mvn -Dinfinispan.test.jgroups.protocol=udp test

Each permutation uses its own report directory, and its own html output file name. This allows you to execute multiple permutations without wiping the results from the previous run. Note that due to the way Maven operates, only one permutation can be executed per mvn command. So automating multiple runs requires shell scripting, or some other execution framework to make multiple calls to Maven.

Running permutations manually or in an IDE

Sometimes you want to run a test using settings other than the defaults (such as UDP for jgroups group tests or the DummyTransactionManager for transaction group tests). This can be achieved by referring to the Maven POM file to figure out which system properties are passed in to the test when doing something different. For example to run a jgroups group test in your IDE using TCP instead of the default UDP, set -Dinfinispan.test.jgroups.protocol=tcp. Or, to use JBoss JTA (Arjuna TM) instead of the DummyTransactionManager in a transaction group test, set -Dinfinispan.test.jta.tm=jbosstm Please refer to the POM file for more properties and permutations.

The Parallel Test Suite

Infinispan runs its unit test suite in parallel; Infinispan tests are often IO rather than processor bound, so executing them in parallel offers significant speed up.

Tips for writing and debugging parallel tests

There are a number of constraints and best practices that need to be followed in order to ensure correctness and keep the execution time to a minimum. If you follow these guidelines you will find your tests are more reliable:

Each test class is run in a single thread
There is no need to synchronize unit test's fixture, as test methods will be run in sequence. However, multiple test classes are executed in parallel.

Each test class must have an unique test name
As a convention, the name of the test should be the FQN of the test class with the org.infinispan prefix removed. For example, given a test class org.infinispan.mypackage.MyTest the name of the test should be mypackage.MyTest. This convention guarantees a unique name.

MyTest.java
package org.infinispan.mypackage;
@Test (testName = "mypackage.MyTest")
public class MyTest { ... }

Use TestCacheManagerFactory.createXyzCacheManager and _never_create managers using new DefaultCacheManager()
This ensures that there are no conflicts on resources e.g. a cluster created by one test won't interfere with a cluster created by another test.

Where possible, extend SingleCacheManagerTestorMultipleCacheManagersTest
Tests inheriting from these template method classes will only create a cache/cluster once for all the test methods, rather than before each method. This helps keep the execution time down.

Never rely onThread.sleep()
When running in heavily threaded environments this will most often not work. For example, if using ASYNC_REPL, don't use a sleep(someValue) and expect the data will be replicated to another cache instance after this delay has elpased. Instead, use a ReplicationListener (check the javadoc for more information). Generally speaking, if you expect something will happen and you don't have a guarantee when, a good approach is to try that expectation in a loop, several times, with an generous (5-10secs) timeout. For example:

while (Systet.currentTimeMillis() - startTime < timeout) {
   if (conditionMeet()) break;
   Thread.sleep(50);
}

Thread.sleep(10) may not work in certain OS/JRE combos (e.g. Windows XP/Sun JRE 1.5)
Use values grater than 10 for these statements, e.g. 50. Otherwise, a System.currentTimeMillis() might return same value when called before and after such a sleep statement.

JMX
For each cache that is create with TestCacheManagerFactory.createXyzCacheManager() the test harness will allocate a unique JMX domain name which can be obtained through CacheManager.getJmxDomain(). This ensures that no JMX collisions will takes place between any tests executed in parallel. If you want to enforce a JMX domain name, this can be done by using one of the TestCacheManagerFactory.createCacheManagerEnforceJmxDomain methods. These methods must be used with care, and you are responsible for ensuring no domain name collisions happen when the parallel suite is executed.

Use obscure words
Insert uncommon or obscure words into the cache that has been generated with a random word generator. In a multi-threaded environment like Infinispan's testsuite, grepping for these words can greatly help the debugging process. You may find this random word generator useful.

Use the test method name as the key
Grab the test method and use it as part of the cached key. You can dynamically grab the test method using code like this:

Thread.currentThread().getStackTrace()(1).getMethodName()

Even though we've tried to reduce them to a min, intermittent failures might still appear from time to time. If you see such failures in existing code please report them in the issue tracker.

JBoss.org Content Archive (Read Only), exported from JBoss Community Documentation Editor at 2020-03-11 09:41:17 UTC, last content change 2012-10-26 09:43:44 UTC.